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DEPARTMENT  OF  HOMELAND  SECURITY 

Better  Planning  and  Assessment  Needed  to  Improve 
Outcomes  for  Complex  Service  Acquisitions 


Why  GAO  Did  This  Study  What  GAO  Found 


The  Department  of  Homeland 
Security  (DHS)  has  relied  on 
service  acquisitions  to  meet  its 
expansive  mission.  In  fiscal  year 
2006,  DHS  spent  $12.7  billion  to 
procure  services.  To  improve 
service  acquisition  outcomes, 
federal  procurement  law 
establishes  a  preference  for  a 
performance-based  approach, 
which  focuses  on  developing 
measurable  outcomes  rather  than 
prescribing  how  contractors  should 
perform  services. 

GAO  was  asked  to  (1)  evaluate  the 
implementation  of  a  performance- 
based  approach  in  the  context  of 
service  acquisitions  for  major, 
complex  investments,  and  (2) 
identify  management  challenges 
that  may  affect  DHS’s  successful 
acquisitions  for  major  investments, 
including  those  using  a 
performance-based  approach. 

GAO  reviewed  judgmentally 
selected  contracts  for  eight  major 
investments  at  three  DHS 
components  totaling  $1.53  billion  in 
fiscal  years  2005  and  2006;  prior 
GAO  and  DHS  Inspector  General 
reviews;  management  documents 
and  plans;  and  related  data, 
including  138  additional  contracts 
for  basic  services. 


What  GAO  Recommends 


DHS  generally  concurred  with 
GAO’s  recommendations  that  DHS 
develop  measurable  standards 
consistently  linked  to  well-defined 
requirements,  evaluate  acquisition 
outcomes  for  major  investments, 
and  improve  data  quality  to  help 
identify  and  assess  contracting 
methods  and  outcomes. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-263. 

For  more  information,  contact  John  Hutton 
(202)512-4841  orhuttonj@gao.gov. 


All  service  contracts  for  the  eight  major,  complex  investments  GAO  reviewed  had 
outcome-oriented  requirements;  however,  four  of  these  contracts  did  not  have 
well-defined  requirements,  a  complete  set  of  measurable  performance 
standards,  or  both.  These  service  contracts  experienced  cost  overruns, 
schedule  delays,  or  did  not  otherwise  meet  performance  expectations.  In 
contrast,  service  contracts  for  the  other  four  investments  GAO  reviewed  had 
well-defined  requirements  linked  to  measurable  performance  standards. 
Contractors  had  begun  work  on  three  of  these  four  contracts  and  performed  within 
budget  meeting  the  standards.  This  finding  is  consistent  with  prior  GAO  work 
on  service  acquisitions,  which  has  highlighted  the  criticality  of  sound 
acquisition  planning  to  develop  well-defined  requirements  and  measurable 
performance  standards  to  achieving  desired  outcomes.  In  the  four  cases  that 
had  negative  outcomes,  program  officials  identified  the  contractor 
performance  weaknesses  through  quality  assurance  surveillance  and  took 
corrective  actions.  Prior  GAO  work  has  found  that  if  acquisitions,  including 
those  that  are  performance-based,  are  not  appropriately  planned,  structured, 
and  monitored,  there  is  an  increased  risk  that  the  government  may  receive 
products  or  services  that  are  over  budget,  delivered  late,  and  of  unacceptable 
quality. 

In  managing  its  service  acquisitions,  including  those  that  are  performance 
based,  DHS  has  faced  workforce  and  oversight  challenges.  Prior  GAO  work 
has  highlighted  the  importance  of  having  the  right  people  with  the  right  skills 
to  achieve  successful  acquisition  outcomes.  Contracts  for  two  major 
investments  with  negative  cost  and  schedule  outcomes  did  not  have  the  staff 
needed  to  adequately  plan  and  execute  the  contracts.  Further,  while 
representatives  for  several  of  the  contracts  GAO  reviewed  indicated  that 
contracting  and  program  staff  worked  well  together,  some  senior  acquisition 
representatives  at  the  component  level  indicated  that  a  lack  of  collaboration 
between  these  key  stakeholders  has  been  a  challenge  when  developing  and 
managing  complex  service  acquisitions.  In  terms  of  oversight,  component 
contracting  and  program  officials  said  they  used  a  perfonnance-based  approach  to 
the  maximum  extent  practicable;  however,  DHS  does  not  have  reliable  data  to 
facilitate  required  reporting,  informed  decisions,  and  analyzing  acquisition 
outcomes.  GAO’s  review  also  found  that  about  half  of  an  additional  138 
contracts  for  basic  services  identified  as  performance-based  did  not  have  any 
of  the  elements  intended  to  foster  good  outcomes:  a  performance  work 
statement,  measurable  performance  standards,  and  a  quality  assurance 
surveillance  plan.  DHS’s  Chief  Procurement  Officer  (CPO) — who  is 
responsible  for  departmentwide  oversight  of  acquisitions — has  several  efforts 
under  way  to  address  some  of  these  workforce  and  oversight  issues.  One 
initiative  is  an  acquisition  oversight  program  that  is  intended  to  assess  (1) 
compliance  with  federal  acquisition  guidance,  (2)  contract  administration,  and 
(3)  business  judgment.  However,  this  oversight  program  has  not  yet  included 
an  evaluation  of  the  outcomes  of  contracting  methods  such  as  performance- 
based  service  acquisition. 
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Abbreviations 


ACE 

CBP 

CPO 

DHS 

FAR 

FEMA 

FPDS-NG 

ICE 

NPI 

OFPP 

SBlnet 


Automated  Commercial  Environment 

Customs  and  Border  Protection 

Chief  Procurement  Officer 

Department  of  Homeland  Security 

Federal  Acquisition  Regulation 

Federal  Emergency  Management  Agency 

Federal  Procurement  Data  System-Next  Generation 

Immigration  and  Customs  Enforcement 

National  Prime  Integration 

Office  of  Federal  Procurement  Policy 

Secure  Border  Initiative  Network 


TSA  Transportation  Security  Administration 

TWIC  Transportation  Worker  Identification  Credential 
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Accountability  *  Integrity  *  Reliability 


United  States  Government  Accountability  Office 
Washington,  DC  20548 


April  22,  2008 

The  Honorable  Joseph  I.  Lieberman 
Chairman 

The  Honorable  Susan  M.  Collins 
Ranking  Member 

Committee  on  Homeland  Security  and  Governmental  Affairs 
United  States  Senate 

The  Honorable  Daniel  K.  Akaka 
Chairman 

The  Honorable  George  V.  Voinovich 
Ranking  Member 

Subcommittee  on  Oversight  of  Government  Management,  the 
Federal  Workforce  and  the  District  of  Columbia 
Committee  on  Homeland  Security  and  Governmental  Affairs 
United  States  Senate 

The  Honorable  Bennie  G.  Thompson 
Chairman 

Committee  on  Homeland  Security 
House  of  Representatives 

In  fiscal  year  2006,  the  Department  of  Homeland  Security  (DHS)  spent 
$12.7  billion  on  services,  representing  more  than  80  percent  of  its  total 
procurement  spending  for  that  year.  DHS  has  relied  on  service  acquisitions 
to  meet  its  expansive  homeland  security  mission.  However,  prior  GAO 
work  has  found  that  if  acquisitions  are  not  appropriately  planned, 
structured,  and  monitored,  there  is  an  increased  risk  that  the  services 
provided  will  not  fulfill  intended  acquisition  outcomes  or,  ultimately,  meet 
government  needs.1 

To  help  improve  service  acquisition  outcomes,  federal  procurement  policy 
calls  for  agencies  to  use  a  performance-based  approach  to  the  maximum 
extent  practicable.  This  approach  includes:  a  performance  work  statement 
that  describes  outcome-oriented  requirements,  measurable  performance 
standards,  and  quality  assurance  surveillance.  These  characteristics,  if 
properly  implemented,  can  help  ensure  that  the  services  meet  cost, 


1  GAO,  Defense  Acquisitions:  Tailored  Approach  Needed  to  Improve  Service  Acquisition 
Outcomes,  GAO-07-20  (Washington,  D.C.:  Nov.  9,  2006). 
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schedule,  and  performance  requirements.  Since  this  concept  was 
introduced  in  the  1990s,  it  has  become  widely  accepted  as  a  sound 
acquisition  approach. 

Despite  its  governmentwide  acceptance,  concerns  have  been  raised  about 
how  well  agencies  are  using  a  performance-based  approach,  particularly 
for  complex  or  major  investments.  Given  the  significant  amount  of  DHS 
spending  on  services  and  its  critical  mission,  you  asked  us  to  (1)  evaluate 
the  implementation  of  a  performance-based  approach  in  the  context  of 
service  acquisitions  for  major,  complex  investments;  and  (2)  identify 
management  challenges  that  may  affect  DHS’s  successful  implementation 
of  service  acquisitions  for  major  investments,  including  those  using  a 
performance-based  approach. 

To  conduct  our  work,  we  focused  on  the  Coast  Guard,  Customs  and 
Border  Protection  (CBP),  Immigrations  and  Custom  Enforcement  (ICE), 
and  the  Transportation  Security  Administration  (TSA) — four  of  the  five 
DHS  components  reporting  the  highest  obligations  for  performance-based 
service  acquisitions  in  fiscal  years  2005  and  2006.2  To  evaluate  DHS’s 
implementation  of  a  performance-based  approach  in  the  context  of  service 
acquisitions  for  major  investments,3  we  judgmentally  selected  eight  major 
investments  at  the  Coast  Guard,  CBP,  and  TSA4 5  based  on  size  and 
complexity,  and  reviewed  actions  for  one  performance-based  service 
contract  associated  with  each  investment.6  These  contracts  had  a 
combined  estimated  value  of  $1.53  billion  and  included  services  provided 
in  the  acquisition  and  sustainment  phases  of  the  investments.6  We  assessed 
these  contracts  against  acquisition  regulations  and  policy  and  interviewed 
contracting  and  program  management  staff.  Performance-based  service 


2We  excluded  the  Federal  Emergency  Management  Agency  (FEMA)  because  of  atypical 
fiscal  year  2006  spending  on  Gulf  Coast  hurricane  relief  efforts. 

3DHS  has  categorized  major  investments  as  Levels  1  and  2  and  Level  3  (information 
technology).  For  the  purposes  of  our  review,  we  focused  on  Level  1  investments.  A  Level  1 
major  investment  is  defined  in  DHS  Management  Directive  1400  as  an  investment  with 
greater  than  $100  million  in  total  acquisition  costs  including  planning,  or  for  information 
technology  investments,  a  life-cycle  cost  greater  than  $200  million. 

4ICE  officials  told  us  that  none  of  their  major  investments  used  a  performance-based 
approach. 

5We  reviewed  either  the  contract  or  selected  orders  on  the  contract. 

6DHS’s  investment  review  process  includes  three  phases:  pre-acquisition,  acquisition,  and 
sustainment.  The  acquisition  phase  includes  systems  development,  and  the  sustainment 
phase  includes  operations  and  maintenance. 
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acquisitions  were  difficult  to  identify  and  were  not  clearly  distinct  from 
other  service  acquisitions  because  they  did  not  always  include  all  required 
elements.  Therefore,  we  reviewed  prior  GAO  reports  and  DHS  Inspector 
General  reports  to  provide  a  broader  context  on  DHS’s  major  service 
acquisitions. 

To  identify  management  challenges  that  may  affect  DHS’s  successful 
implementation  of  service  acquisitions  for  major  investments,  we  analyzed 
management  documents  and  plans;  staffing  data;  and  oversight 
mechanisms,  including  the  Federal  Procurement  Data  System-Next 
Generation  (FPDS-NG),  the  governmentwide  database  for  procurement 
spending.  From  FPDS-NG,  we  judgmentally  selected  138  contracts  for 
primarily  basic  services  coded  as  performance-based  and  awarded  in  fiscal 
years  2005  or  2006  at  Coast  Guard,  CBP,  ICE,  and  TSA  with  obligations 
greater  than  $1  million  dollars  and  asked  the  components  to  verify 
whether  these  contracts  included  the  required  performance-based 
elements.  Finally,  we  interviewed  contracting  and  program  management 
staff  and  DHS  Chief  Procurement  Office  (CPO)  representatives  and  met 
with  representatives  from  the  Office  of  Management  and  Budget’s  Office 
of  Federal  Procurement  Policy  (OFPP).  For  more  information  on  our 
scope  and  methodology,  see  appendix  I. 

We  conducted  this  performance  audit  from  January  2007  to  April  2008  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 


Results  in  Brief 


All  service  contracts  for  the  eight  major,  complex  investments  we 
reviewed  had  outcome-oriented  requirements,  but  the  requirements  were 
not  always  well-defined.  Four  of  the  eight  contracts  we  reviewed  did  not 
have  well-defined  requirements,  or  a  complete  set  of  measurable 
performance  standards,  or  both  at  the  time  of  contract  award  or  start  of 
work.  These  service  contracts  experienced  cost  overruns,  schedule  delays, 
or  did  not  otherwise  meet  performance  expectations.  In  contrast,  service 
contracts  for  the  other  four  investments  had  well-defined  requirements 
linked  to  measurable  performance  standards.  Contractors  had  begun  work 
on  three  of  these  four  contracts  and  performed  within  budget  meeting  the 
standards.  This  finding  is  consistent  with  our  prior  work  on  service 
contracting,  which  has  highlighted  the  criticality  of  sound  acquisition 
planning  to  develop  stable,  well-defined  requirements  and  measurable 
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performance  standards  to  achieving  desired  outcomes.  Components 
conducted  quality  assurance  surveillance  for  the  contracts  we  reviewed. 
For  the  four  contracts  that  had  negative  outcomes,  surveillance  helped 
DHS  components  to  identify  contractor  performance  weaknesses  and 
corrective  actions  were  taken.  Prior  GAO  work  has  found  that  if 
acquisitions,  including  those  that  are  performance-based,  are  not 
appropriately  planned,  structured,  and  monitored,  there  is  an  increased 
risk  that  the  government  may  receive  products  or  services  that  are  over 
budget,  delivered  late,  and  of  unacceptable  quality. 

In  managing  its  service  acquisitions,  including  those  that  are  performance- 
based,  DHS  has  faced  workforce  and  oversight  challenges.  Our  prior  work 
has  highlighted  the  importance  of  having  the  right  people  with  the  right 
skills  to  achieve  successful  acquisition  outcomes.  Contracts  for  two  of  the 
eight  major  investments  we  reviewed  that  had  negative  cost  or  schedule 
outcomes  did  not  have  the  staff  needed  to  adequately  plan  and  execute  the 
contracts.  Further,  while  representatives  for  several  of  the  contracts  we 
studied  indicated  contracting  and  program  staff  worked  together 
effectively,  at  the  component  level,  senior  acquisition  representatives 
indicated  that  a  lack  of  collaboration  between  the  program  and 
contracting  offices  in  general  has  been  challenging.  In  terms  of  oversight, 
DHS  component  contracting  and  program  staff  told  us  they  used  a 
performance-based  approach  to  the  maximum  extent  practicable; 
however,  the  department  does  not  have  reliable  data  to  facilitate  reporting 
or  perform  management  assessments  of  these  acquisitions.  Moreover, 
while  the  CPO  has  implemented  a  departmentwide  acquisition  oversight 
program,  the  CPO  has  not  performed  management  assessments  of 
performance-based  service  acquisitions.  Our  review  of  an  additional  138 
contracts  for  primarily  basic  services  found  that  about  half  of  the 
contracts  identified  as  performance-based  in  FPDS-NG  did  not  have  any  of 
the  three  elements:  a  performance  work  statement,  measurable 
performance  standards,  and  a  quality  assurance  surveillance  plan,  which 
are  intended  to  foster  good  outcomes.  Inaccurate  data  limit  DHS’s 
visibility  over  service  acquisitions  and  the  department’s  ability  to  make 
informed  acquisition  management  decisions.  DHS’s  CPO  has  several 
efforts  under  way  to  address  departmentwide  workforce  and  oversight 
challenges.  One  initiative  is  an  acquisition  oversight  program  that  is 
intended  to  assess  (1)  compliance  with  federal  acquisition  guidance,  (2) 
contract  administration,  and  (3)  business  judgment.  However,  this 
oversight  program  has  not  yet  included  an  evaluation  of  the  outcomes  of 
contracting  methods  such  as  performance-based  service  acquisition. 
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To  increase  DHS’s  ability  to  improve  outcomes  for  its  service  acquisitions, 
including  those  that  are  performance-based,  we  are  recommending  that 
the  Secretary  of  Homeland  Security  take  several  actions.  These  actions 
include  improving  acquisition  planning  for  requirements  for  major, 
complex  investments  to  ensure  they  are  well-defined,  and  developing 
consistently  measurable  performance  standards  linked  to  those 
requirements;  systematically  evaluating  the  outcomes  of  major 
investments  and  relevant  contracting  methods  at  a  departmentwide  level; 
and  improving  the  quality  of  FPDS-NG  data  so  that  DHS  can  more 
accurately  identify  and  assess  the  use  and  outcomes  of  various  contracting 
methods.  In  written  comments  on  a  draft  of  this  report,  DHS  generally 
concurred  with  our  recommendations  and  provided  some  information  on 
efforts  under  way  to  improve  acquisition  management.  The  department’s 
comments  are  reprinted  in  appendix  II.  OMB  did  not  comment  on  the 
findings  or  conclusions  of  this  report.  DHS  and  OMB  each  provided 
technical  comments,  which  we  incorporated  as  appropriate  and  where 
supporting  documentation  was  provided. 


Background 


Over  the  last  decade,  the  use  of  federal  service  contracting  has  increased 
and  now  accounts  for  over  60  percent  of  federal  procurement  dollars 
spent  annually.  Prior  GAO  work  has  found  that  if  acquisitions  are  not 
appropriately  planned,  structured,  and  monitored,  there  is  an  increased 
risk  that  the  services  provided  will  not  fulfill  intended  acquisition 
outcomes  or,  ultimately,  meet  agency  needs. 

A  performance-based  approach,  if  successfully  implemented,  can  help 
improve  service  acquisition  outcomes.  This  concept — introduced  during 
the  1990s — represented  a  shift  in  emphasis  from  specifying  methods  in 
which  contracted  work  should  be  performed  to  specifying  acquisition 
outcomes.  In  2000,  federal  procurement  law  established  a  performance- 
based  approach  as  the  preferred  acquisition  method  for  services.7  The  FAR 
requires  all  performance-based  service  acquisitions  to  include: 


'Floyd  D.  Spence  National  Defense  Authorization  Act  for  Fiscal  Year  2001,  Pub.  L.  No.  106- 
398  §  821(a)  (2000)  required  that  the  Federal  Acquisition  Regulation  (FAR)  be  revised  to 
establish  a  preference  for  the  use  of  a  performance-based  approach  in  the  acquisition  of 
services,  which  was  done  in  FAR  37.102(a),  providing  that  performance-based  acquisition 
is  the  preferred  method  for  acquiring  services  and  generally  is  to  be  used  to  the  maximum 
extent  practicable. 
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•  a  performance  work  statement  that  describes  outcome-oriented 
requirements  in  terms  of  results  required  rather  than  the  methods  of 
performance  of  the  work; 

•  measurable  performance  standards  describing  how  to  measure 
contractor  performance  in  terms  of  quality,  timeliness,  and  quantity; 
and 

•  the  method  of  assessing  contract  performance  against  performance 
standards,  commonly  accomplished  through  the  use  of  a  quality 
assurance  surveillance  plan.8 

In  addition,  federal  procurement  law  established  a  preference  for  using 
firm  fixed-price  contracts  or  task  orders — where  a  specified  price  is  paid 
regardless  of  the  contractor’s  incurred  costs — when  using  a  performance- 
based  approach  for  service  acquisitions.9  A  performance-based  acquisition 
approach  also  calls  for  greater  collaboration  among  procurement  and 
program  representatives  throughout  the  contract  period  and  often  requires 
more  time  for  acquisition  planning  and  management  than  other 
contracting  methods.  Agencies  may  also  involve  the  contractor  in  the 
multi-functional  team  responsible  for  developing  performance-based 
acquisition  planning  documents  to  leverage  the  contractor’s  technical 
expertise.  Including  stakeholders  with  varied  knowledge  and  skills  enables 
acquisition  teams  to  determine  at  the  earliest  point  possible  that  all 
aspects  of  the  acquisition  are  necessary,  executable,  and  tailored  to  the 
transaction’s  risk  level.  Regardless  of  the  contracting  method,  focusing  on 
outcomes  and  collaboration  among  multiple  stakeholders  in  the 
contracting  process  has  been  acknowledged  as  sound  contract 
management.  Other  factors,  such  as  pressure  to  get  programs  up  and 
running,  additional  external  requirements,  and  technological  challenges 
also  impact  the  ability  to  achieve  good  acquisition  outcomes. 

A  1998  OFPP  study  on  performance-based  contracts — based  largely  on 
contracts  for  basic  services,  such  as  janitorial  or  maintenance  services — 
showed  that  a  number  of  anticipated  benefits  had  been  achieved:  reduced 
acquisition  costs,  increased  competition  for  contracts,  and  improved 


SFAR  37.601;  FAR  37.602(b);  FAR  37.604.  A  fourth  element,  performance  incentives,  is 
required  where  appropriate. 

9Floyd  D.  Spence  National  Defense  Authorization  Act  for  Fiscal  Year  2001,  Pub.  L.  No.  106- 
398,  §  821(a)  (2000);  FAR  37.102(a). 
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contractor  performance.10  However,  our  prior  work  and  the  work  of  others 
has  found  that  implementing  a  performance-based  approach  is  often  more 
difficult  for  complex  acquisitions  because  agencies  begin  with 
requirements  that  are  less  stable,  making  it  difficult  to  establish 
measurable  outcomes.  OFPP  has  noted  in  policy  that  certain  types  of 
services,  such  as  research  and  development,  may  not  lend  themselves  to 
outcome-oriented  requirements.11  CPO  representatives  also  have  noted 
that  defining  outcome-oriented  requirements  and  measurable  performance 
standards  may  be  more  challenging  for  certain  types  of  services,  such  as 
research  and  development  or  professional  and  management  support 
services.  Further,  complex  service  contracts,  such  as  those  for  information 
technology,  may  need  to  have  requirements  and  performance  standards 
continually  refined  throughout  the  life-cycle  of  the  acquisition  for  a 
contractor  to  deliver  a  valuable  service  over  an  extended  period  of  time. 

To  encourage  agencies  to  apply  a  performance-based  approach  to  service 
acquisitions,  the  Office  of  Management  and  Budget  established 
governmentwide  performance  targets:  20  percent  of  eligible  service 
contract  dollars  for  fiscal  year  2002,  40  percent  for  fiscal  year  2005  and 
fiscal  year  2006,  45  percent  for  fiscal  year  2007,  and  50  percent  for  fiscal 
year  2008. 

In  2007,  the  congressionally  mandated  Acquisition  Advisory  Panel12 
reported  on  the  implementation  of  a  performance-based  approach  and 
concluded  that  agencies  were  not  clearly  defining  requirements,  not 
identifying  meaningful  quality  measures,  not  effectively  managing  the 
contract,  and  had  limited  data  to  make  informed  decisions.  The  panel 
made  recommendations  to  improve  application  of  the  performance-based 
approach  in  three  areas:  goal  setting  and  management,  guidance  for 
acquisition  staff,  and  data  quality.13  For  example,  the  panel  recommended 


10Office  of  Management  and  Budget,  Office  of  Federal  Procurement  Policy,  A  Report  on  the 
Performance-Based  Service  Contracting  Pilot  Project ,  May  1998. 

nOffice  of  Federal  Procurement  Policy  Memorandum,  “Increasing  the  Use  of  Performance- 
based  Service  Acquisitions,”  September  7,  2004. 

12Services  Acquisition  Reform  Act  of  2003,  Title  XIV,  National  Defense  Authorization  Act 
for  Fiscal  Year  2004,  Pub.  L.  No.  108-136,  §  1423  (2003);  National  Defense  Authorization  Act 
for  Fiscal  Year  2006,  Pub.  L.  No.  109-163,  §  843  (2006)  (extended  the  deadline  for  the 
Panel’s  report  by  6  months). 

13Report  of  the  Acquisition  Advisory  Panel  to  the  Office  of  Federal  Procurement  Policy  and 
the  United  States  Congress,  January  2007. 


Page  7 


GAO-08-263  DHS  Service  Acquisitions 


that  the  Office  of  Management  and  Budget  adjust  the  govemmentwide 
target  to  reflect  individual  agency  assessments  and  plans. 

In  May  2007,  OFPP  issued  a  memo  directing  agencies,  at  a  minimum,  to 
meet  targets  established  and  report  on  them  in  their  management  plans.  In 
response,  DHS’s  CPO,  who  is  responsible  for  creating  departmentwide 
policies  and  processes  for  managing  and  overseeing  the  acquisition 
function,  established  a  performance-based  target  of  25  percent  for  fiscal 
year  2007,  increasing  to  40  percent  by  fiscal  year  2010,  that  was  included  in 
DHS’s  Performance-Based  Management  Plan.  Meeting  these  targets  is  a 
responsibility  shared  between  the  CPO  and  the  component  heads,  in 
accordance  with  DHS’  system  of  dual  accountability  for  managing  the 
acquisition  function.14 


Definition  of 
Requirements  and 
Performance 
Standards  Influenced 
Contract  Outcomes 


Requirements  definition  and  performance  standards  influenced  outcomes 
for  performance-based  contracts  associated  with  eight  major,  complex 
investments.  While  all  eight  contracts  for  the  investments  we  reviewed  had 
outcome-oriented  requirements,  the  requirements  were  not  always  well- 
defined.16  Further,  contracts  for  half  of  the  investments  did  not  have  a 
complete  set  of  measurable  performance  standards.  Major  investments 
with  contracts  that  did  not  have  well-defined  requirements  or  complete 
measurable  performance  standards  at  the  time  of  contract  award  or  start 
of  work  experienced  either  cost  overruns,  schedule  delays,  or  did  not 
otherwise  meet  performance  expectations.  Conversely,  contracts  with 
well-defined  requirements  linked  to  measurable  performance  standards 
delivered  results  within  budget  and  provided  quality  service.  This  finding 
is  consistent  with  our  body  of  work  on  service  contracting,  which  has 


Responsibility  for  the  acquisition  function  at  DHS  is  shared  between  the  CPO  and  each 
DHS  component  head.  Eight  DHS  components  have  internal  procurement  offices  with  a 
Head  of  Contracting  Activity  (HCA)  who  reports  directly  to  the  component  head  and  is 
accountable  to  the  CPO.  The  eight  components  are:  Coast  Guard,  CBP,  FEMA,  Federal  Law 
Enforcement  Training  Center,  ICE,  Office  of  Procurement  Operations,  Secret  Service,  and 
TSA.  The  HCA  for  each  component  has  overall  responsibility  for  the  day-to-day 
management  of  the  component’s  acquisition  function.  See  GAO,  Department  of  Homeland 
Security:  Progress  and  Challenges  in  Implementing  the  Department’s  Acquisition 
Oversight  Plan,  GAO-07-900  (Washington,  D.C.:  June  2007). 

15In  using  the  performance-based  approach,  sound  contracting  practices  dictate  that 
required  contract  outcomes  or  requirements  be  well-defined,  providing  clear  descriptions 
of  results  to  be  achieved.  FAR  2.101  specifically  provides  that  a  performance  work 
statement  for  performance-based  acquisitions  describe  the  required  results  in  clear, 
specific,  and  objective  terms  with  measurable  outcomes. 
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emphasized  the  importance  of  sound  acquisition  planning  to  develop  well- 
defined  requirements  and  measurable  performance  standards  to  achieve 
desired  results.  Table  1  summarizes  our  analysis  of  the  requirements, 
performance  standards,  and  outcomes  for  the  eight  performance-based 
contracts  we  reviewed.  Component  program  staff  conducted  surveillance 
for  these  contracts,  and  in  the  four  cases  that  had  negative  outcomes, 
identified  less  than  expected  performance  and  took  action. 


Table  1:  Key  Characteristics  of  Eight  Performance-Based  Service  Contracts 

Major  investment  by 
component 

Service 

Well-defined 

requirements8 

Measurable 

performance 

standards” 

Outcomes 

Coast  Guard 

Response  Boat 

Medium 

Research,  analysis,  and 
financial  and  information 
management 

• 

• 

Contractor  submitted  all  required  documentation 
on  time;  met  project  management  quality 
standards;  and  maintained  electronic  archiving 
and  restoration  standards. 

Customs  and  Border  Protection 

Automated 

Commercial 

Environment 

T rade  systems  software 
development  (task  order 
23) 

O 

® 

Costs  increased  by  40  percent  ($21.1  million). 
More  than  a  year  behind  schedule;  unplanned 
software  redesign. 

National  Prime 
Integration 

Maintenance  of 
equipment  used  at 
border  crossings, 
airports,  and  seaports 

• 

® 

Costs  increased  by  53  percent  ($24  million). 
Maintenance  wait  times  were  longer  than 
planned. 

SBI  net 

Project  28  border 
surveillance  systems 
development  and  fielding 

o 

® 

DHS  rejected  initial  acceptance  of  Project  28. 

The  project  was  delayed  8  months  with  final 
acceptance  in  February  2008.  DHS  noted  that 
the  contractor  met  the  requirements,  but  the 
project  did  not  fully  meet  DHS’s  needs  and  the 
technology  will  not  be  replicated  in  future  SBInef 
development. 

Transportation  Security  Administration 

Electronic  Baggage 
Screening  Program 

Maintenance  for 
explosive  trace  detection 
machines 

• 

• 

Contractor  exceeded  the  performance  standard 
for  machine  downtime  with  a  score  1  hour  less 
than  required  and  operated  at  cost  through  the 
second  quarter  of  fiscal  year  2007. 

Screening  Partnership 
Program 

Passenger  screening 
services  at  one  airport 

• 

• 

Contractor  exceeded  most  performance 
standards;  for  example:  threat  detection 
performance  and  false  alarm  rates  exceeded  the 
quality  standards.  Contractor  had  cost  underrun 
of  2.2  percent  ($677,000). 
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Major  investment  by 
component 

Service 

Well-defined 

requirements8 

Measurable 

performance 

standards” 

Outcomes 

Secure  Flight 

Maintaining  database 
used  to  screen  airline 
passenger  data 

• 

® 

Initial  contractor  planning  reports  were 
inadequate;  system  experienced  operational 
downtime;  surveillance  reports  identified  poor 
contractor  performance.  Contractor  generally  met 
time  frames  and  delivered  within  budget. 

Transportation  Worker  Issuing  identification  Outcomes  not  available  at  the  time  of  our  review. 

Identification  credentials  to  maritime  •  • 

Credential  workers 


Legend:  •  contract  met  or  mostly  met  the  criteria;  ®  contract  partially  met  the  criteria;  O  contract  did 
not  meet  the  criteria 

Source:  GAO  analysis. 

aWell-defined  requirements  should  provide  clear  descriptions  of  results  to  be  achieved  at  the  time  of 
the  award  or  start  of  work  and  primary  requirements  should  not  change  substantially  following 
contract  award. 

bThe  set  of  measurable  performance  standards  for  a  contract  enables  the  government  to  assess  all 
aspects  of  the  contractor’s  work  in  terms  of  quality,  timeliness,  and  quantity.  The  contract's 
performance  standards  are  also  linked  to  the  requirements. 


Contracts  That  Lacked 
Stable,  Well-Defined 
Requirements,  Complete 
Measurable  Performance 
Standards,  or  Both 
Experienced  Negative 
Outcomes 


Although  all  contracts  we  reviewed  for  the  eight  major,  complex 
investments  had  outcome-oriented  requirements,  contracts  for  four 
investments — two  for  systems  development  and  two  for  operations  and 
maintenance — did  not  have  stable  and  well-defined  requirements,  a  full 
complement  of  measurable  performance  standards,  or  both.  These  four 
contracts  experienced  schedule  delays,  cost  increases,  or  other  negative 
outcomes.  Prior  GAO  and  DHS  Inspector  General  work  on  other  DHS 
major  investments  have  similarly  found  that  when  requirements  were  not 
well-defined,  performance  did  not  meet  expectations.  Our  work  has  found 
that  performance-based  acquisitions  must  be  appropriately  planned  and 
structured  to  minimize  the  risk  of  the  government  receiving  services  that 
are  over  cost  estimates,  delivered  late,  and  of  unacceptable  quality.16 
Specifically,  we  have  emphasized  the  importance  of  clearly  defined 
requirements  to  achieving  desired  results  and  measurable  performance 
standards  to  ensuring  control  and  accountability.17 


ltJGAO,  Coast  Guard:  Observations  on  the  Fiscal  Year  2008  Budget,  Performance, 
Reorganization,  and  Related  Challenges,  GAO-07-489T  (Washington  D.C.:  Apr.  18,  2007). 

17GAO-07-20;  and  GAO,  Coast  Guard:  Status  of  Efforts  to  Improve  Deepwater  Program 
Management  and  Address  Operational  Challenges,  GAO-07-575T  (Washington  D.C.:  Mar. 
8,  2007). 
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Systems  Development  Efforts 


Two  CBP  systems  development  contracts  lacked  both  well-defined 
requirements  and  measurable  performance  standards  prior  to  the  start  of 
work.  The  first — ACE  Task  Order  23  project,  a  trade  software 
modernization  effort  under  a  letter  contract18 — was  originally  estimated  to 
cost  $52.7  million  over  a  period  of  approximately  17  months.  DHS 
contracting  representatives  told  us  that  some  technical  requirements  were 
not  fully  defined  within  the  original  scope  of  work,  and  some  new 
software  requirements  were  added  after  contract  award.  Because  many 
requirements  were  not  fully  defined  in  advance  of  the  contract’s  award, 
the  program  was  unable  to  establish  clear,  measurable  performance 
standards  and  valid  cost  or  schedule  baselines  for  assessing  contractor 
performance.  The  need  to  redefine  contract  requirements  for  the  ACE 
program  contributed  to  significant  schedule  delays  and  cost  increases.19 
ACE  Task  Order  23  was  completed  at  the  end  of  October  2007  with  a 
project  cost  increase  of  approximately  $21.1  million,  or  40  percent,  over 
the  original  estimate.  As  of  April  2008,  DHS  reported  that  some  portions  of 
the  work  were  delayed  to  better  define  requirements,  and  the  final  portion 
of  the  software  was  projected  to  be  completed  in  June  2009,  about  26 
months  later  than  planned. 

The  second,  CBP’s  Secure  Border  Initiative  Network  (SBIwef)  Project  28 
contract,  was  intended  to  help  secure  a  section  of  the  United  States- 
Mexico  border  using  a  surveillance  system.  Project  28  was  planned  to  be 
SBlnet  's  proof  of  concept,  as  well  as  the  first  increment  of  the  fielded 
SB! net  system.  CBP  awarded  the  Project  28  contract  before  the  overall 
SBI net  operational  requirements  and  system  specifications  were  finalized. 
Additional  design  requirements  for  equipment  installation  were  added 
after  contract  award.  DHS’s  Inspector  General  reported  in  February 
2007 — more  than  3  months  after  the  Project  28  contract  was  awarded — 
that  CBP  had  not  properly  defined  SBIne/’s  operational  requirements  and 
needed  to  do  so  quickly  to  avoid  rework  of  the  contractor’s  systems 
engineering.  Several  performance  standards  were  not  clearly  defined  to 
isolate  the  contractor’s  performance  from  that  of  CBP  employees,  making 
it  difficult  to  determine  whether  any  problems  were  due  to  the 
contractor’s  system  design,  CBP  employees,  or  both.  Other  performance 
standards  were  difficult  to  measure  because  CBP  did  not  plan  a  controlled 


18A  letter  contract  authorizes  a  contractor  to  begin  work  and  incur  costs  before  reaching  a 
final  agreement  on  contract  terms  and  conditions,  including  price. 

19GAO,  Information  Technology:  Improvements  for  Acquisition  of  Customs  Trade 
Processing  System  Continue,  but  Further  Efforts  Needed  to  Avoid  More  Cost,  and 
Schedule  Shortfalls,  GAO-08-46  (Washington,  D.C.:  Oct.  25,  2007). 
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Operations  and  Maintenance 
Efforts 


operational  test,  which  is  a  test  conducted  in  a  separate,  instrumented  test 
area  that  resembles  the  border  environment.  CBP  planned  to  conduct 
operational  testing,  which  is  less  rigorous,  after  the  system  was  fully 
accepted.  As  a  result,  it  was  not  clear  how  CBP  intended  to  measure 
compliance  with  the  Project  28  standard  for  probability  of  detecting 
persons  attempting  to  illegally  cross  the  border.  Another  standard  stated 
that  the  acceptable  level  of  contractor  performance  would  be  determined 
in  the  future.  Project  28,  planned  as  an  8-month,  $20  million  firm  fixed- 
price  effort,20  did  not  meet  the  June  2007  delivery  date,  and  in  August  2007 
CBP  notified  the  contractor  that  it  did  not  approve  delivery  because  the 
system  failed  the  initial  acceptance  test.  DHS  conditionally  accepted  the 
system  in  December  2007  and  fully  accepted  it  in  February  2008.  DHS 
noted  that  the  contractor  met  the  requirements;  however,  Project  28 
resulted  in  a  product  that  did  not  fully  meet  user  needs  and  its  design  will 
not  be  used  as  a  basis  for  future  SBIrtef  development.21  In  addition,  DHS 
officials  have  stated  that  much  of  the  Project  28  system  will  be  replaced  by 
new  equipment  and  software. 

Performance-based  contracts  for  two  other  major  investments — CBP’s 
National  Prime  Integration  (NPI)  contract  and  a  TSA  Secure  Flight 
contract — included  well-defined  requirements  but  did  not  have  a  full 
complement  of  measurable  performance  standards.  These  operations  and 
maintenance  contracts  had  less  than  expected  contractor  performance  or 
significant  cost  increases. 

CBP’s  NPI  contract  for  maintenance  of  inspection,  detection,  and 
surveillance  devices  at  border  crossings,  airports,  and  seaports  included 
well-defined  requirements  for  conducting  equipment  deployment, 
maintenance,  and  replacement  as  well  as  engineering  and  logistics 
support.  For  example,  the  requirements  for  equipment  corrective 
maintenance  specify  the  contractor  shall  conduct  maintenance,  including 
parts  and  labor,  as  a  result  of  equipment  or  system  failure.  However,  the 
contract  did  not  incorporate  performance  standards  for  post-maintenance 
inspections  and  other  standards  needed  to  fully  assess  the  quality  of 
equipment  maintenance  performed  by  the  contractor.  CBP  representatives 
acknowledged  the  need  for  additional  measurable  performance  standards 


20The  contractor  testified  in  October  2007  before  subcommittees  of  the  House  Homeland 
Security  Committee,  that  it  spent  approximately  double  the  original  contract  amount. 
Because  this  is  a  firm  fixed  price  contract,  absent  changes  or  delays  created  by  the 
government,  CBP  is  not  responsible  for  these  additional  costs. 

21GAO,  Secure  Border  Initiative:  Observations  on  the  Importance  of  Applying  Lessons 
Learned  to  Future  Projects,  GAO-08-508T  (Washington  D.C.:  Feb.  27,  2008). 
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Prior  Reviews  Have  Similar 
Findings 


addressing  quality  of  equipment  maintenance  to  help  ensure  good 
contractor  performance.  While  CBP  does  not  plan  to  incorporate 
additional  performance  measures  until  September  2008,  CBP  revised  the 
contract  documentation  in  December  2007  to  emphasize  program 
management.  Regarding  contract  outcomes,  during  the  last  year  of  the 
contract,  costs  were  about  $24  million,  or  53  percent,  higher  than  original 
estimates,  in  part  because  of  increases  in  the  quantity  of  equipment  being 
supported.  Customer  wait  times  have  been  much  longer  than  expected  for 
equipment  maintenance,  and  wait  time  standards  have  been  revised  to 
between  24  and  96  hours  depending  on  the  type  of  equipment.  Average 
wait  times  from  September  2006  to  September  2007  were  between  118  and 
106  hours,  not  meeting  the  revised  standard. 

A  TSA  Secure  Flight  contract  for  operations  and  maintenance  of  the 
federal  terrorist  watchlist  database,  which  is  used  to  screen  airline 
passenger  data,  also  experienced  less  than  expected  outcomes.  The 
contract  we  reviewed  contained  well-defined  requirements  and 
measurable  performance  standards  for  operation,  maintenance,  repair, 
and  upgrade  of  the  system.  For  example,  the  requirements  specify  that  the 
contractor  shall  provide  on-site  technical  and  management  support. 
Although  the  contract  had  some  measurable  performance  standards,  the 
contract  did  not  have  the  full  complement  of  standards  needed  to  monitor 
performance,  according  to  program  representatives.  For  example,  one 
standard  required  no  operational  system  downtime  and  the  contractor  was 
not  always  able  to  meet  this  standard.  After  we  reviewed  the  performance 
standards,  program  representatives  subsequently  incorporated  some 
additional,  improved  standards  to  provide  increased  visibility  over  how 
well  the  contractor  responds  to  system  outages.  These  standards  are 
designed  to  measure  the  time  required  to  switch  from  the  primary  to  the 
secondary  system  for  planned  and  unplanned  system  outages. 

Additionally,  according  to  program  and  contracting  representatives,  the 
contractor  initially  delivered  inadequate  plans  for  staffing  and  operations, 
and  did  not  always  meet  the  operational  system  downtime  requirement, 
but  was  within  budget  and  generally  met  established  time  frames. 

The  inability  to  manage  requirements  and  meet  cost,  schedule,  and 
performance  objectives  is  not  limited  to  the  components  and  systems  we 
reviewed.  Several  prior  GAO  and  DHS  Inspector  General  reviews  of  major 
DHS  investments  using  a  performance-based  approach  point  to  such 
shortcomings.  For  example: 
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•  In  June  2007,  we  reported  that  DHS’s  performance-based  contract  for  a 
departmentwide  financial  management  system,  eMerge2,  did  not  have 
clear  or  complete  requirements.22  Consequently,  the  program 
experienced  schedule  delays  and  less  than  acceptable  contractor 
performance  and  was  ultimately  terminated  after  a  $52  million 
investment. 

•  In  March  2007,  we  reported  that  the  Coast  Guard’s  performance-based 
contract  for  replacing  or  modernizing  its  fleet  of  vessels  and  aircraft, 
Deepwater,  had  requirements  that  were  set  at  unrealistic  levels  and 
then  were  frequently  changed.23  This  resulted  in  cost  escalation, 
schedule  delays,  and  reduced  contractor  accountability.  The  DHS 
Inspector  General  has  also  indicated  that  numerous  opportunities  exist 
for  DHS  to  make  better  use  of  sound  practices,  such  as  well-defined 
requirements.  The  DHS  Inspector  General  recently  concluded  that 
Deepwater  had  poorly  defined  requirements  and  inadequate  oversight 
that  contributed  to  ineffective  or  inefficient  results  and  increased 
costs.24 

•  In  February  2006,  DHS’s  Inspector  General  reported  that  TSA’s 
Information  Technology  Managed  Services  contract  had  unclear 
requirements  that  contributed  to  negative  outcomes,  including  an  85 
percent  increase  to  nearly  $834  million  in  DHS’s  3-year  cost  estimate 
and  a  failure  to  deliver  expected  information  technology  capabilities.26 


Contracts  with  Well- 
Defined  Requirements  and 
Measurable  Performance 
Standards  Experienced 
Positive  Outcomes 


Contracts  for  four  other  major  investments  we  reviewed — one  at  the  Coast 
Guard  and  three  at  TSA — had  well-defined  requirements  and  measurable 
performance  standards.  These  requirements  did  not  significantly  change  in 
scope  after  contract  award.  Contracts  for  program  management  support 
for  the  Response  Boat-Medium  program,  maintenance  of  the  Electronic 
Baggage  Screening  Program,  and  passenger  and  baggage  screening  for  the 


22GAO,  Homeland  Security:  Departmentwide  Integrated  Financial  Management  Systems 
Remain  a  Challenge,  GAO-07-536  (Washington,  D.C.:  June  21,  2007). 

23GAO-07-575T. 

24Department  of  Homeland  Security  Inspector  General,  Major  Management  Challenges 
Facing  the  Department  of  Homeland  Security,  OIG-08-11  (January  2008). 

25Department  of  Homeland  Security  Inspector  General,  Transportation  Security 
Administration’s  Information  Technology  Managed  Services  Contract,  OIG-06-23 
(February  2006). 


Page  14 


GAO-08-263  DHS  Service  Acquisitions 


Screening  Partnership  Program  met  the  performance  standards  and  were 
within  budget.  We  were  unable  to  evaluate  outcomes  for  the  fourth 
contract  associated  with  the  Transportation  Worker  Identification 
Credential  (TWIC)  program  because  key  work  under  the  contract  was 
delayed. 

Program  management  support  and  related  services  we  reviewed  for  the 
Coast  Guard’s  Response  Boat-Medium  investment  had  well-defined 
requirements,  and  two  of  three  performance  standards  were  measurable. 
The  contracted  services  included  research  and  project  support,  financial 
and  information  management,  and  Web  site  support.  The  requirements  for 
research  and  project  support  clearly  stated  the  contractor  was  to  provide 
project  planning  and  execution  support;  documentation,  database,  and 
internet  support;  and  administrative  support.  The  performance  standards 
were  linked  directly  to  the  requirements.  The  two  measurable  standards 
were  for  timely  submission  of  documents  and  electronic  records  keeping. 
The  Coast  Guard  did  not  indicate  how  it  would  measure  contractor 
performance  for  the  third  standard  addressing  research  analysis  and 
advisory  services.  Coast  Guard  representatives  said  the  contractor  had 
fulfilled  the  requirements  within  budget  and  on  time,  but  did  not  document 
contractor  performance. 

A  maintenance  contract  for  TSA’s  Electronic  Baggage  Screening  Program 
also  had  well-defined  program  management  and  technical  requirements. 
For  example,  the  requirements  for  maintenance  clearly  stated  the 
contractor  should  perform  preventative  and  corrective  maintenance  on 
detection  machines,  providing  specific  definitions  for  each  of  these  tasks. 
The  contract  also  had  measurable  performance  standards  linked  directly 
to  the  technical  requirements.  Specifically,  program  representatives  used 
the  ‘mean  downtime’  performance  standard  to  calculate  the  average 
number  of  hours  a  machine  was  unable  to  perform  its  mission  during  a 
16-hour  day  as  a  means  of  evaluating  how  quickly  the  contractor  restored 
the  detection  equipment  to  operation.  For  fiscal  years  2006  through  2007, 
the  contractor  met  or  exceeded  the  mean  downtime  performance 
standard,  which  has  become  more  rigorous  each  year.  Additionally,  the 
contractor  has  stayed  within  the  budget  established  in  the  firm  fixed-price 
contract.  The  current  maintenance  contract  improved  TSA’s  ability  to 
manage  costs.  Previously,  DHS’s  Inspector  General  found  TSA’s  initial 
electronic  baggage  screening  maintenance  contracts  did  not  follow  sound 
contracting  practices.  The  Inspector  General  recommended  TSA  take 
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action  to  control  costs  by  modifying  the  type  of  contract  used.26  TSA 
subsequently  withdrew  the  initial  maintenance  contract  and  issued  four 
new  contracts,  one  of  which  we  reviewed,  for  maintenance  of  detection 
machines  at  over  400  airports  in  the  United  States. 

Contracted  security  services  at  the  San  Francisco  International  Airport  for 
the  Screening  Partnership  Program  also  had  well-defined  requirements, 
and  all  measurable  performance  standards  corresponded  to  contract 
requirements.  This  was  an  improvement  from  our  prior  reviews  of  TSA’s 
Screening  Partnership  Program,  which  found  that  TSA  had  not  finalized 
performance-based  planning  documentation,  including  developing 
measurable  performance  standards.27  The  contract  we  reviewed  contained 
well-defined  requirements  for  gate,  checkpoint,  and  baggage  screening; 
basic  preventative  and  corrective  maintenance;  initial,  recurrent,  and 
remedial  training;  and  recruiting  and  supervising  screeners.  For  example, 
the  requirements  for  gate,  checkpoint,  and  baggage  screening  services 
clearly  stated  the  contractor  should  use  technology  and  staff  to  prevent 
prohibited  items  from  entering  sterile  areas  of  the  airport  and  should  work 
to  minimize  customer  complaints  while  addressing  in  a  timely  manner  any 
complaints  received.  The  performance  standards  assessed  how  often 
screeners  could  successfully  detect  test  images  of  prohibited  items  in 
checked  baggage;  the  percentage  of  audited  records  and  inspected 
equipment,  property,  and  materials  that  were  well-kept,  operational,  and 
recorded  on  maintenance  logs;  and  whether  all  new  hires  received  the 
required  training  before  assuming  their  screening  responsibilities.  In  terms 
of  expected  outcomes,  the  contractor  achieved  a  2.2  percent  cost 
underrun  during  the  first  5  months  of  the  contract  and  exceeded  most 
requirements. 

While  the  worker  enrollment  services  contract  for  TSA’s  TWIC  program 
for  a  tamper-resistant  biometric  credential  for  maritime  workers  included 
well-defined  requirements  and  measurable  performance  standards,  we 
were  unable  to  review  the  contract’s  outcomes  as  key  work  under  the 


^Department  of  Homeland  Security  Inspector  General,  Evaluation  of  TSA’s  Contract  for 
the  Installation  and  Maintenance  of  Explosive  Detection  Equipment  at  United  States 
Airports,  OIG-04-44  (September  2004). 

2 'GAO,  Aviation  Security:  Preliminary  Observations  on  TSA’s  Progress  to  Allow  Airports 
to  Use  Private  Passenger  and  Baggage  Screening  Services,  GAO-05-126  (Washington,  D.C.: 
Nov.  19,  2004);  and  GAO,  Aviation  Security:  Progress  Made  to  Set  Up  Program  Using 
Private-Sector  Airport  Screeners,  but  More  Work  Rem  ains,  GAO-06-166  (Washington,  D.C.: 
Mar.  31,  2006). 
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contract  was  delayed.28  Prior  GAO  work  found  that  during  the  TWIC 
program’s  prototype  phase,  the  program  contract  costs  doubled  due  to 
expanded  requirements  after  contract  award.29  The  contract  we  reviewed 
contained  well-defined  requirements  for  developing  enrollment  centers; 
providing  information  technology  maintenance  services;  and  setting  up 
help  desk  services  for  TSA  employees,  maritime  workers,  and  other  TWIC 
stakeholders.  For  example,  the  requirements  for  enrollment  centers 
clearly  stated  the  contractor  was  to  provide  maintenance  services; 
infrastructure  support  such  as  work  stations;  and  operations, 
management,  and  administrative  support.  Moreover,  the  contract’s 
performance  standards  were  measurable  and  linked  to  the  requirements. 
For  example,  the  program  developed  performance  standards  assessing  the 
amount  of  time  the  contractor  took  to  respond  to  and  resolve  each  caller’s 
issue  and  to  enroll  maritime  workers  in  the  program.  By  using  measurable 
performance  standards  linked  to  the  contract’s  requirements,  TSA  has 
improved  its  ability  to  identify  and  address  performance  issues  and  avoid 
negative  outcomes. 


Quality  Assurance 
Surveillance  Identified 
Less  Than  Expected 
Performance 


Agencies  are  required  to  ensure  that  contractors  are  providing  timely  and 
quality  services  and  mitigate  contractor  performance  problems.  The  FAR 
indicates  that  surveillance  plans  provide  for  contract  quality  assurance  by 
specifying  contractor  work  requiring  surveillance  and  the  method  of 
surveillance.30  While  not  all  of  the  contracts  we  reviewed  had  a  quality 
assurance  surveillance  plan  in  place  at  contract  award,  in  all  cases  DHS 
components  monitored  and  assessed  contractor  performance  against  the 
contract  performance  standards.  In  the  four  cases  where  we  found  poor 
contract  outcomes,  program  staff  identified  and  documented  less  than 
expected  performance  and  took  action  as  shown  in  table  2. 


28TSA  awarded  a  contract  in  January  2007  to  issue  biometric  credentials  to  maritime 
workers.  TSA  officials  told  us  it  took  more  time  than  originally  planned  to  bring  the  TWIC 
prototype  enrollment  system  into  alignment  with  new  government  security  standards. 
During  the  course  of  our  review,  the  schedule  to  initiate  work  was  delayed  from  March  to 
October  2007. 

29GAO,  Transportation  Security:  DHS  Should  Address  Key  Challenges  before 
Implementing  the  Transportation  Worker  Identification  Credential  Program,  GAO-06-982 
(Washington,  D.C.:  Sept.  29,  2006). 

30FAR  37.604  and  46.401. 
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Table  2:  Less  Than  Expected  Performance  Identified  through  Surveillance  and  Action  Taken 

Major  investment  by 
component 

Service 

Performance 

Action  taken 

Customs  and  Border  Protection 

Automated  Commercial 
Environment  Task  Order 
23 

Developing  trade  systems 
software 

Costs  increased  by  40  percent 
($21.1  million)  and  contractor  did 
not  achieve  project  milestones 
according  to  schedule 

Contractor  did  not  earn  full 
incentive  fees 

National  Prime 

Integration 

Maintaining  equipment  used 
at  border  crossings,  airports, 
and  seaports 

Longer  than  expected 
maintenance  wait  times 

Contractor  did  not  earn  full  award 
fees 

SBI  net  Project  28 

Developing  and  fielding  border 
surveillance  systems 

System  failed  the  initial 
performance  test  and  project  was 
delayed  8  months  with  final 
acceptance  in  February  2008 

Contractor  did  not  receive  20 
percent  of  contract  payment  and 
DHS  did  not  initially  accept  the 
system 

Transportation  Security  Administration 

Secure  Flight 

Maintaining  database  used  to 
screen  airline  passenger  data 

Inadequacies  identified  in  initial 
contractor  planning  reports 
provided  to  TSA;  operational 
downtime;  and  other  forms  of  less 
than  expected  performance 

Contractor  did  not  earn  full  award 
fees  and  DHS  worked  with 
contractor  to  increase  staffing 
levels 

Source:  GAO  analysis. 

In  two  of  these  cases — ACE  and  Secure  Flight — program  representatives 
identified  less  than  expected  performance  through  the  use  of  a  quality 
assurance  surveillance  plan.  In  contrast,  the  NPI  program  office  told  us 
they  did  not  have  a  quality  assurance  surveillance  plan  for  the  first  year  of 
the  contract,  but  instead  relied  on  award  fee  plan  criteria  to  assess 
contractor  performance.  The  SBI net  program  office  monitored  Project  28’s 
execution  through  other  means,  such  as  program  reviews,  technical 
interchange  meetings,  status  and  issue  reporting,  and  observation  of 
contractor  testing.  Our  prior  work  has  found  that  if  performance-based 
acquisitions  are  not  appropriately  planned  and  structured,  including 
surveillance  planning,  there  is  an  increased  risk  that  the  government  may 
receive  products  or  services  that  are  over  budget,  delivered  late,  and  of 
unacceptable  quality.31 


31GAO-07-489T. 
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DHS  Has  Faced 
Workforce  and 
Oversight  Challenges 
in  Managing  Its 
Service  Acquisitions 


Insufficient  workforce  and  limited  oversight  of  acquisition  outcomes  has 
presented  significant  challenges  for  DHS  in  implementing  its  service 
acquisitions,  including  those  that  use  a  performance-based  approach. 
Having  adequate  staff  who  collaborate  across  functions,  and  reliable  data 
are  among  the  key  success  factors  that  we  have  identified  for  achieving 
intended  outcomes  for  all  types  of  service  acquisitions.32  However,  DHS 
continues  to  lack  contracting  and  program  staff  with  the  expertise  needed 
to  adequately  plan  or  monitor  contractor  execution  of  requirements,  and 
some  component  acquisition  representatives  indicated  a  lack  of 
collaboration  between  the  program  and  contracting  offices.  In  terms  of 
oversight,  inaccurate  data  has  limited  departmentwide  visibility  into  DHS’s 
service  acquisitions,  including  those  that  are  performance-based. 


Sufficient  and  Experienced 
Staff  Are  Needed  to 
Ensure  Successful  Service 
Acquisitions 


Our  prior  work  has  highlighted  the  importance  of  having  the  right  people 
with  the  right  skills  to  achieve  successful  acquisition  outcomes.33  However, 
DHS  has  not  fully  defined  the  types  of  positions  or  numbers  of  staff  for 
each  position  for  its  acquisition  workforce.  CPO  representatives  identified 
acquisition  staff  shortages  as  one  of  the  primary  obstacles  to  successful 
acquisitions,  including  those  that  are  performance-based.  CPO 
representatives  have  established  departmentwide  interim  staffing  goals 
addressing  minimum  and  optimal  staffing  levels  for  contract  specialists  for 
each  component.  As  of  February  2008,  CPO  representatives  reported  that 
approximately  61  percent  of  the  minimum  required  staff,  and  38  percent  of 
the  optimal  level  of  contract  specialists,  were  in  place. 


Insufficient  contracting  and  program  office  staff  was  a  challenge  for  two 
of  the  investments  we  reviewed — SBI net  and  ACE.  In  these  two  cases, 
contracts  were  not  on  schedule  or  contractor  performance  was  otherwise 
less  than  expected.  In  November  2006 — the  month  following  SBI  net 
Project  28’s  contract  award — the  DHS  Inspector  General  reported  that 
DHS  did  not  have  the  capacity  needed  to  effectively  plan,  oversee,  and 
execute  the  SBIwef  investment;  administer  its  contracts;  and  control  costs 
and  schedule.34  For  example,  SBIwef  had  a  shortage  of  engineers,  logistical 
personnel,  contracting  officers,  and  cost  analysts.  In  January  2007,  DHS’s 
Investment  Review  Board  confirmed  there  were  critical  staffing  shortages 


32GAO,  Framework  for  Assessing  the  Acquisition  Function  at  Federal  Agencies, 

GAO-05-2 18G  (Washington,  D.C.:  Sept.  2005),  and  GAO-07-20. 

33GAO-07-20. 

34Department  of  Homeland  Security  Inspector  General,  Risk  Management  Advisory  for  the 
SBInet  Program  Initiation,  OIG-07-07  (November  2006). 
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in  the  SBIne/  investment,  including  Project  28.  In  February  2007,  we 
reported  that  SBI net  representatives  expressed  concern  about  finding  an 
adequate  number  of  program  office  staff  with  the  right  expertise.  We 
subsequently  reported  in  February  2008  that  the  SBI  net  program  office 
staffing  level  had  reached  305  positions  out  of  a  staffing  goal  of  470  by  the 
end  of  fiscal  year  2008.35  CBP  representatives  expressed  concerns  that 
staffing  shortfalls  could  affect  the  agency’s  capacity  to  provide  adequate 
contractor  oversight.  ACE  program  officials  also  indicated  they  were 
significantly  understaffed,  which  prevented  them  from  fully  defining  the 
requirements  for  ACE  Task  Order  23.  Officials  told  us  the  investment  had 
less  than  10  percent  of  the  staff  they  needed  at  the  time  the  requirements 
were  being  defined. 

In  its  fiscal  year  2007  departmentwide  performance-based  service 
acquisition  management  plan,  DHS  also  emphasized  the  importance  of 
filling  program  and  contracting  vacancies  with  staff  experienced  in 
developing  and  implementing  the  required  elements  of  performance-based 
acquisitions.  Contracting  and  program  management  staff  for  TSA’s 
Screening  Partnership  and  Electronic  Baggage  Screening  programs  stated 
that  their  previous  experience  helped  them  to  develop  well-defined 
requirements  and  measurable  performance  standards.  In  these  cases, 
contracts  were  on  budget  and  contract  performance  met  or  exceeded  the 
government’s  requirements.  Representatives  from  Coast  Guard,  CBP,  and 
TSA  noted  success  in  supplementing  expertise  through  acquisition  support 
offices,  also  known  as  centers  of  excellence,  to  improve  acquisition 
planning.  Component  representatives  told  us  this  support  could  help 
acquisition  staff  improve  the  quality  of  contract  documents,  including 
performance  work  statements.  These  support  offices  can  serve  as  a  force 
multiplier,  allowing  their  high-demand  skill-sets  to  be  used  across 
contracts  and  helping  to  improve  contracting  outcomes  across  the 
component. 

According  to  CPO  representatives,  hiring  knowledgeable  acquisition  staff 
has  been  difficult  due  to  a  limited  pool  of  qualified  acquisition 
professionals  and  intense  competition  between  the  government  and  the 
private  sector  for  those  professionals.36  The  CPO  has  the  following  ongoing 
departmentwide  initiatives  to  recruit,  train,  and  define  its  acquisition 
workforce  that  have  the  potential  to  better  position  DHS  to  address 
acquisition  workforce  challenges.  These  include: 


3BGAO-08-508T. 

36We  have  ongoing  work  on  DHS’s  acquisition  workforce  and  plan  to  report  on  these  issues 
in  the  final  product  for  that  engagement. 
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•  centralize  recruiting  activities  across  the  department; 

•  institute  an  acquisition  intern  program; 

•  employ  direct  hire  authority37  and  rehire  annuitants;38 

•  reemphasize  training  opportunities  and  establishing  a  centralized 
acquisition  workforce  training  fund;  and 

•  conduct  workforce  planning. 

The  need  for  improved  collaboration  among  the  acquisition  workforce  is 
also  a  recurrent  theme  in  our  work  on  acquisition  management.  We  have 
noted  that  acquisition  groups  should  have  procedures  in  place  that 
empower  staff  to  collaborate  when  procuring  goods  and  services  and  have 
controls  and  incentives  to  ensure  collaboration  occurs.39  In  particular, 
leading  organizations  generally  employ  a  multifunctional  approach 
including  contracting,  finance,  legal,  and  other  participants  as  needed  to 
help  develop  cost-effective  acquisition  approaches  and  help  ensure 
financial  accountability.  When  requirements  are  dynamic,  as  was  the  case 
for  some  of  the  investments  we  studied,  there  is  a  greater  need  for  more 
sophisticated  collaboration — a  point  echoed  by  CBP  and  TSA 
representatives  who  emphasize  the  importance  of  early  collaboration  for 
complex  acquisitions.  OFPP  guidance  also  emphasizes  that  performance- 
based  service  acquisition  is  a  collective  responsibility  involving 
representatives  from  budget,  technical,  contracting,  logistics,  legal,  and 
program  offices. 

For  the  eight  major  investments  we  reviewed,  program  and  contracting 
offices  jointly  approved  the  acquisition  plans,  and  representatives  for 
several  of  the  contracts  we  studied  indicated  they  worked  together 
effectively.  However,  at  the  component  level,  senior  acquisition  staff  at 


3'Services  Acquisition  Reform  Act  of  2003,  Div.  A,  Title  XIV,  National  Defense  Authorization 
Act  for  Fiscal  Year  2004,  Pub.  L.  No.  108-136,  §  1413  (2003)  allowed  federal  agencies  to 
exercise  direct  hire  authority  in  accordance  with  regulations  issued  by  the  Office  of 
Personnel  Management.  This  authority  was  extended  to  September  30,  2012,  by  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2008,  Pub.  L.  No.  110-181,  Div.  A,  Title 
VIII,  §  853  (2008). 

38General  Services  Administration  Modernization  Act,  Pub.  L.  No.  109-313,  §  4  (2006) 
granted  agency  heads  authority,  in  consultation  with  the  OPM  and  OFPP,  to  approve 
reemployment  of  annuitants  in  fields  related  to  acquisition  management. 

39GAO-05-218G. 
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TSA  and  CBP  indicated  that  collaboration  between  contracting  and 
program  offices  in  general  has  been  challenging.  For  example,  they  cited 
poor  timing  when  bringing  together  key  stakeholders  in  the  contracting 
process,  and  in  some  cases,  the  program  offices’  preference  for 
prescriptive  requirements  rather  than  a  more  outcome-oriented  approach. 
Coast  Guard  and  TSA  acquisition  representatives  we  spoke  with  indicated 
they  have  ongoing  efforts  to  promote  greater  collaboration. 


Limited  Data  Inhibits 
DHS’s  Ability  to  Oversee 
Its  Service  Acquisitions 


While  DHS  contracting  and  program  representatives  told  us  they  use  a 
performance-based  approach  to  the  maximum  extent  practicable,  DHS 
does  not  have  reliable  data,  either  from  FPDS-NG  or  at  a  departmentwide 
level,  to  systematically  monitor  or  evaluate  service  acquisitions,  including 
those  that  are  performance-based  acquisitions.  Reliable  data  are  essential 
to  overseeing  and  assessing  the  implementation  of  contracting 
approaches,  acquisition  outcomes,  and  making  informed  management 
decisions. 


Our  review  of  138  contracts  at  the  Coast  Guard,  CBP,  ICE,  and  TSA  coded 
in  FPDS-NG  as  performance-based  illustrates  the  difficulty  in  identifying 
such  contracts.40  According  to  contracting  representatives  at  the  Coast 
Guard,  CBP,  ICE,  and  TSA,  only  42,  about  30  percent,  of  these  contracts 
could  be  confirmed  as  containing  the  required  performance-based 
elements — a  performance  work  statement,  measurable  performance 
standards,  and  a  method  of  assessing  contractor  performance  against 
performance  standards.  About  18  percent  had  some  but  not  all  of  the 
required  performance-based  acquisition  elements,  and  about  51  percent — 
totaling  about  $347.3  million — had  none  of  the  required  elements  (see 
table  3).  This  analysis  indicates  that  the  FPDS-NG  data  are  not  reliable  for 
reporting  on  the  performance  target  for  eligible  service  obligations.  For 
example,  in  fiscal  year  2006,  DHS  reported  awarding  performance-based 
contracts  for  21  percent  of  eligible  service  contract  dollars.  However,  due 
to  unreliable  data,  DHS  is  likely  farther  away  from  meeting  the 
governmentwide  target  than  previously  reported.  In  addition,  in  July  2006 
OFPP  requested  all  agencies  to  report  in  their  5-year  management  plan  on 
the  use  of  performance-based  contracts  by  service  types — ranging  from 
basic,  such  as  janitorial  and  landscaping,  to  complex,  such  as  information 
technology  or  systems  development.  The  Acquisition  Advisory  Panel  and 


40The  138  contracts  were  awarded  in  fiscal  years  2005  or  2006  with  obligations  greater  than 
$1  million,  totaling  about  $1.3  billion. 


Page  22 


GAO-08-263  DHS  Service  Acquisitions 


DHS’s  CPO  also  have  raised  concerns  regarding  the  accuracy  of  the 
performance-based  designation  in  FPDS-NG.  The  Acquisition  Advisory 
Panel  reported  from  its  review  at  10  federal  agencies  that  42  percent  of  the 
performance-based  contracts  it  reviewed  had  been  incorrectly  coded.41 


Table  3:  Review  of  Performance-Based  Elements  on  Selected  Contracts 


Performance-based 

elements 

Coast 

Guard 

Customs  and 
Border 
Protection 

Immigration  and 
Customs 
Enforcement 

Transportation 

Security 

Administration 

Total 

contracts 

Percent  of 
total  contracts 

All  elements 

18 

3 

0 

21 

42 

30.4 

Some  elements 

16 

0 

5 

4 

25 

18.1 

No  elements 

20 

5 

34 

12 

71 

51.5 

Total 

54 

8 

39 

37 

138 

100 

Source:  GAO  analysis  of  DHS  review  of  138  contracts  coded  as  performance-based  in  FPDS-NG. 


According  to  DHS  contracting  representatives,  contracts  were  miscoded 
as  performance-based  for  several  reasons.  Contracting  staff  may  have 
coded  contracts  as  performance-based  without  the  presence  of  the 
required  elements.  Contracting  staff  also  may  not  have  followed  data 
validation  procedures.  Finally,  there  are  technical  FPDS-NG  issues.  For 
example,  DHS  contracting  and  program  staff  said  FPDS-NG  automatically 
applies  the  same  designation  to  all  orders  issued  under  the  master 
contract  although  some  individual  orders  may  or  may  not  be  performance- 
based.42  While  DHS  officials  acknowledged  this  can  occur,  they  stated  that 
the  data  field  in  FPDS-NG  could  still  be  edited  for  orders  issued  under  the 
master  contract  and  that  it  was  incumbent  on  DHS  contracting 
representatives  to  review  the  designation  for  every  contract  action  to 
ensure  they  are  coded  correctly. 

Inaccurate  federal  procurement  data  is  a  long-standing  governmentwide 
concern.  Our  prior  work  and  the  work  of  the  General  Services 
Administration’s  Inspector  General  have  noted  issues  with  the  accuracy 


41Report  of  the  Acquisition  Advisory  Panel  to  the  Office  of  Federal  Procurement  Policy  and 
the  United  States  Congress,  January  2007. 

42Known  as  Indefinite  Delivery  Indefinite  Quantity  (IDIQ)  contracts,  these  contracts 
provide  for  an  indefinite  quantity  of  supplies  or  services  within  stated  limits  during  a  fixed 
period  of  time.  The  government  places  orders  for  individual  requirements. 
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and  completeness  of  FPDS  and  FPDS-NG  data.43  The  Office  of 
Management  and  Budget  has  stressed  the  importance  of  submitting  timely 
and  accurate  procurement  data  to  FPDS-NG  and  issued  memos  on  this 
topic  in  August  2004  and  March  2007. 

Accurate  FPDS-NG  data  could  facilitate  the  CPO’s  departmentwide 
oversight  of  service  acquisitions,  including  those  that  are  performance- 
based.  For  example,  the  selection  of  a  contract  type  is  the  principal  means 
agencies  have  for  allocating  financial  risk  between  the  government  and  the 
contractor.  For  performance-based  services,  federal  law  establishes  a 
preference  for  using  firm  fixed-price  as  opposed  to  cost  reimbursable  or 
time  and  materials  contracts.44  In  addition,  the  CPO  has  instructed 
components  to  avoid  using  time-and-material  contracts  for  services.  We 
reviewed  the  contract  type  for  the  42  contracts  DHS  confirmed  were 
performance-based.  Our  analysis  showed  that  DHS  awarded  about  70 
percent  of  these  42  contracts  as  fixed-price;  these  actions  represented 
about  42  percent  of  total  obligations  for  these  awards.  About  37  percent  of 
total  obligations  were  awarded  as  cost-reimbursable,  and  7  percent  were 
awarded  as  time  and  materials  contracts  (see  fig.  1). 


43GAO,  Reliability  of  Federal  Procurement  Data,  GAO-04-295R  (Washington,  D.C.:  Dec.  30, 
2003);  GAO,  Improvements  Needed  to  the  Federal  Procurement  Data  System-Next. 
Generation,  GAO-05-960R  (Washington,  D.C.:  Sept.  27,  2005);  and  General  Services 
Administration  Inspector  General,  Review  of  the  Federal  Procurement  Data  System.-Next. 
Generation  (FPDS-NG),  Report  Number  A040127/0/T/F06016  (March  2006). 

44For  fixed  price  contracts,  a  specified  price  is  paid  regardless  of  the  contractor’s  costs, 
minimizing  the  financial  risk  to  the  government.  For  cost  reimbursable  and  time  and 
materials  contacts,  the  government  generally  assumes  the  risk  of  cost  overruns.  For 
example,  cost  reimbursable  contracts  provide  for  the  government  to  pay  reasonable, 
allowable  and  allocable  costs  incurred  by  the  contractor  up  to  the  contract’s  price  ceiling. 
Time  and  materials  contracts  provide  for  acquiring  services  on  the  basis  of  fixed  costs  for 
labor  hours  and  materials  at  the  stated  contract  ceiling  price.  In  time  and  materials 
contracts,  if  services  delivered  do  not  meet  contract  requirements  and  the  government 
exercises  its  right  to  have  the  contractor  correct  the  deficiencies,  the  government  pays  the 
additional  labor  and  material  costs  to  do  so,  excluding  profit. 
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Figure  1:  Percent  of  Contracts  and  Obligations  by  Contract  Type  for  42  DHS 
Performance-Based  Service  Acquisitions  (Fiscal  Years  2005  and  2006) 


Percent 

80 


Contracts 

Obligations 

Source:  GAO  analysis  of  DHS  confirmed  performance-based  service  acquisitions. 

““Other"  includes  contracts  for  which  the  contract  type  field  in  FDPS-NG  was  blank,  designated  as 
being  none  of  the  contract  types  listed  within  FPDS-NG,  or  coded  with  more  than  one  contract  type. 

At  a  departmentwide  level,  CPO  representatives  responsible  for 
acquisition  oversight  indicated  they  have  not  conducted  systematic 
assessments  including  costs,  benefits,  and  other  outcomes  of  a 
performance-based  approach.  To  improve  the  implementation  of 
performance-based  acquisitions,  CPO  representatives  established  a  work 
group  in  May  2006  to  leverage  knowledge  among  DHS  components.  They 
also  noted  that  they  are  working  with  OFPP  to  develop  a  best  practices 
guide  on  measurable  performance  standards  and  to  gather  good  examples 
of  performance-based  contracts.  In  addition,  the  CPO  has  implemented  a 
departmentwide  acquisition  oversight  program  that  is  intended  to  assess 
(1)  compliance  with  federal  acquisition  guidance,  (2)  contract 
administration,  and  (3)  business  judgment.  This  program  was  designed 
with  the  flexibility  to  address  specific  procurement  issues,  such  as 
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performance-based  service  acquisitions,  and  is  based  on  a  series  of 
component-level  reviews.46  However,  this  oversight  program  has  not  yet 
included  an  evaluation  of  the  outcomes  of  this  acquisition  method. 


Conclusions 


As  part  of  its  expansive  homeland  security  mission,  DHS  spends  billions  of 
dollars  on  service  acquisitions  for  critical  trade,  transportation,  and  border 
security  investments.  Consistent  with  federal  procurement  policy,  DHS 
has  emphasized  a  performance-based  approach  to  improve  service 
acquisition  outcomes.  However,  in  keeping  with  our  prior  findings,  DHS’s 
designation  of  a  service  acquisition  as  performance-based  was  not  as 
relevant  as  the  underlying  contract  conditions.  Sound  acquisition 
practices,  such  as  clearly  defining  requirements;  establishing 
complementary  measurable  performance  standards;  and  planning  and 
conducting  surveillance  in  order  to  take  corrective  actions  are  all 
hallmarks  of  successful  service  acquisitions.  In  the  cases  we  reviewed,  as 
well  as  in  prior  findings  lacking  these  key  elements,  DHS  did  not  always 
achieve  successful  acquisition  outcomes.  To  improve  DHS’s  acquisition 
management,  the  CPO  is  taking  steps  to  address  departmentwide 
workforce  and  oversight  needs.  However,  improved  data  accuracy  and 
systematic  evaluation  of  contracting  methods  and  outcomes  are  also  key 
to  successful  oversight  efforts.  Continued  emphasis  on  these  areas  is 
required  to  prevent  less  than  expected  acquisition  outcomes  for  the 
department’s  critical  mission. 


Recommendations  for 
Executive  Action 

•  routinely  assess  requirements  for  major,  complex  investments  to 
ensure  they  are  well-defined  and  develop  consistently  measurable 
standards  linked  to  those  requirements; 

•  at  a  departmentwide  level,  systematically  evaluate  the  outcomes  of 
major  investments  and  relevant  contracting  methods;  and 


To  increase  DHS’s  ability  to  achieve  improved  outcomes  for  its  service 
acquisitions,  including  those  that  are  performance-based,  we  recommend 
that  the  Secretary  of  Homeland  Security  implement  the  following  three 
actions: 


4BGAO-07-900. 
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•  continuously  improve  the  quality  of  FPDS-NG  data  to  facilitate  the 
ability  to  accurately  identify  and  assess  the  use  and  outcomes  of 
various  contracting  methods. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  DHS  and  OMB  for  review  and 
comment.  In  written  comments,  DHS  generally  concurred  with  our 
findings  and  recommendations  and  provided  some  information  on  efforts 
under  way  to  improve  acquisition  management.  The  department’s 
comments  are  reprinted  in  appendix  II.  OMB  did  not  comment  on  the 
findings  or  conclusions  of  this  report.  DHS  and  OMB  each  provided 
technical  comments,  which  we  incorporated  as  appropriate  and  where 
supporting  documentation  was  provided. 

In  response  to  our  recommendations  that  DHS  (1)  routinely  assess 
requirements  for  major,  complex  investments  to  ensure  they  are  well- 
defined,  and  develop  consistently  measurable  standards  linked  to  those 
requirements,  and  (2)  systematically  evaluate  the  outcomes  of  major 
investments  and  relevant  contracting  methods,  DHS  stated  that  the  CPO  is 
working  to  strengthen  acquisition  and  procurement.  For  example,  DHS 
noted  that  the  CPO  is  revising  the  investment  review  process;  reviewing 
major  programs  and  investments;  and  building  the  capability  to  manage 
complex  efforts  by  ensuring  program  offices  are  properly  structured  and 
staffed  with  the  right  people  and  skills.  To  assist  in  accomplishing  these 
goals,  DHS  noted  that  the  CPO  has  established  a  new  division  to  provide 
oversight  and  support  for  acquisition  programs.  We  have  ongoing  work  at 
DHS  reviewing  the  results  of  DHS’s  oversight  of  major  acquisitions  and 
plan  to  report  on  these  initiatives  in  the  final  product  for  that  engagement. 
Improving  acquisition  management  has  been  an  ongoing  challenge  since 
the  department  was  established  and  requires  sustained  management 
attention.  However,  DHS’s  response  did  not  address  how  the  CPO’s 
process  and  organizational  changes  will  impact  component-level 
management  and  assessment  of  complex  acquisitions  to  improve 
outcomes. 


Concerning  the  third  recommendation,  that  DHS  continuously  improve  the 
quality  of  FPDS-NG  data  to  facilitate  the  ability  to  accurately  identify  and 
assess  the  use  and  outcomes  of  various  contracting  methods,  DHS  stated 
that  as  part  of  the  CPO’s  oversight  reviews,  the  accuracy  of  the  FPDS-NG 
data  is  validated  for  the  review  sample,  including  whether  contracts  have 
been  properly  coded  as  performance-based.  DHS  also  added  that  the  CPO 
is  an  active  member  of  the  OFPP  group  working  to  improve  FPDS-NG  and 
has  established  a  governance  board  whereby  the  CPO  reaches  out  to  DHS 


Page  27 


GAO-08-263  DHS  Service  Acquisitions 


components  to  improve  data  collection.  While  these  initiatives  may  be 
steps  in  the  right  direction,  DHS’s  response  did  not  present  the  results  of 
the  oversight  reviews,  or  state  how  these  coordination  efforts  address  the 
causes  of  miscoding  or  how  they  will  improve  the  quality  of  FPDS-NG  data 
in  the  future. 

DHS  also  noted  other  initiatives  to  improve  the  acquisition  workforce, 
including  working  to  obtain  qualified  acquisition  professionals  and 
conducting  staffing  studies  to  better  define  the  department’s  needs.  We 
also  have  ongoing  work  at  DHS  on  the  acquisition  workforce  and  plan  to 
report  on  the  results  of  DHS’s  efforts  in  the  final  product  for  that 
engagement. 


As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report,  we  plan  no  further  distribution  for  30  days  from  the  report 
date.  At  that  time,  we  will  send  copies  of  this  report  to  the  Secretary  of 
Homeland  Security,  the  Director  of  the  Office  of  Management  and  Budget, 
and  other  interested  congressional  committees.  We  will  also  make  copies 
available  to  others  upon  request.  In  addition,  this  report  will  also  be 
available  at  no  charge  on  GAO’s  Web  site  at  http://www.gao.gov. 

If  you  have  questions  about  this  report  or  need  additional  information, 
please  contact  me  at  (202)  512-4841  or  huttonj@gao.gov.  Contact  points 
for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found 
on  the  last  page  of  this  report. 


John  P.  Hutton,  Director 
Acquisition  and  Sourcing  Management 
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Appendix  I:  Scope  and  Methodology 


To  evaluate  selected  Department  of  Homeland  Security  (DHS) 
components’  implementation  of  a  performance-based  approach  in  the 
context  of  service  acquisitions  for  major  investments,  we  conducted  a 
detailed  file  review  of  eight  judgmentally  selected  performance-based 
contracts  for  major,  complex  investments  with  a  total  value  greater  than 
$100  million.1  To  identify  the  components  with  the  highest  obligations 
using  a  performance-based  approach  in  fiscal  years  2005  and  2006,  we 
used  the  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG) 
and  selected  four  components — Coast  Guard,  Customs  and  Border 
Protection  (CBP),  Immigration  and  Customs  Enforcement  (ICE),  and  the 
Transportation  Security  Administration  (TSA).2  We  excluded  the  Federal 
Emergency  Management  Agency  (FEMA)  from  our  review  because  of 
atypical  fiscal  year  2006  spending  on  Gulf  Coast  hurricane  relief  efforts. 

To  identify  contracts  for  major  investments,  we  utilized  several  different 
approaches.3  Three  issues  required  us  to  modify  our  plans — the  lack  of 
information  in  FPDS-NG,  inaccurate  DHS  contracting  data  on  major 
investments,  and  inaccurate  information  provided  by  one  DHS  component. 
Because  the  FPDS-NG  major  program  field  was  typically  blank,  we  were 
unable  to  use  this  field  to  identify  contracts  associated  with  DHS  major 
investments.  As  a  result,  we  asked  DHS  to  provide  us  with  a  list  of 
performance-based  contracts  associated  with  the  four  components’  major 
investments.  DHS  provided  a  list  including  76  contracts  for  16  major 
investments.  Due  to  data  inaccuracies  related  to  the  contract  numbers  on 
DHS’s  list,  we  were  only  able  to  identify  performance-based  contracts 
associated  with  two  major  investments.  We  selected  one  contract  from 
CBP’s  Automated  Commercial  Environment  and  one  from  TSA’s  Secure 
Flight  that  had  the  highest  obligations.  To  obtain  additional  contracts  to 
review,  we  met  with  CBP  and  TSA  contracting  representatives  and  they 
identified  five  other  major  investments  with  performance-based  contracts, 
and  we  included  one  from  each  investment  in  our  review.  Coast  Guard 


DIIS  has  categorized  major  investments  as  Levels  1  and  2  and  Level  3  (information 
technology).  For  the  purposes  of  our  review,  we  focused  on  Level  1  investments.  A  Level  1 
major  investment  is  defined  in  DHS  Management  Directive  1400  as  an  investment  with 
greater  than  $100  million  in  total  acquisition  costs  including  planning,  or  for  information 
technology  investments,  a  life-cycle  cost  greater  than  $200  million. 

"During  our  review  TSA  was  exempt  from  the  FAR  and  followed  the  Acquisition 
Management  System,  developed  by  the  Federal  Aviation  Administration,  which  included 
acquisition  policy  providing  that  service  contracts  should  incorporate  performance-based 
contracting  methods. 

3We  selected  eight  major  investments  and  reviewed  either  a  performance-based  contract  or 
selected  orders  on  a  performance-based  contract  associated  with  each  investment. 
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contracting  representatives  told  us  none  of  their  major  investments  used 
performance-based  contracts.  However,  we  randomly  selected  and 
reviewed  four  contracts  and  seven  orders  at  the  Coast  Guard  from  DHS’s 
list  of  major  investments,  and  found  that  five  of  them  were  performance- 
based.  Of  these,  we  selected  three  performance-based  orders  from  one 
contract  for  one  major  Coast  Guard  investment — the  Response  Boat- 
Medium — to  include  in  our  review.  Lastly,  ICE  contracting  representatives 
also  reported  that  none  of  their  major  investments  used  a  performance- 
based  approach  and  provided  us  with  their  best  examples  of  a 
performance-based  contract.  We  reviewed  these  examples  and  found  that 
none  of  them  were  associated  with  a  major  investment.  As  a  result,  we 
excluded  ICE  from  our  contract  review. 

For  the  selected  contracts,  with  a  combined  estimated  value  of  $1.53 
billion,  we  reviewed  documentation,  including  acquisition  plans, 
performance  work  statements  and  statements  of  objectives,  quality 
assurance  surveillance  plans,  and  government  or  contractor  performance 
reports  where  available.  Performance-based  service  acquisitions  were 
difficult  to  identify  and  were  not  clearly  distinct  from  other  service 
acquisitions  because  they  did  not  always  include  all  required  elements. 
Therefore,  we  reviewed  prior  GAO  and  DHS  Inspector  General  reports  to 
provide  a  broader  context  for  service  acquisition  and  other  DHS  major 
investments.  We  reviewed  public  laws,  federal  and  agency  acquisition 
regulations,  and  the  Office  of  Management  and  Budget’s  Office  of  Federal 
Procurement  Policy  (OFPP)  memoranda  and  best  practices.  We  also 
interviewed  procurement  and  program  representatives  to  determine  the 
extent  to  which  the  performance-based  planning  process  influenced 
expected  outcomes. 

To  identify  management  challenges  that  may  affect  DHS’s  successful 
implementation  of  service  acquisitions  for  major  investments,  including  those 
that  use  a  performance-based  approach,  we  analyzed  management 
documents  and  plans;  staffing  data;  and  oversight  mechanisms,  including 
the  Federal  Procurement  Data  System-Next  Generation  (FPDS-NG),  the 
governmentwide  database  for  procurement  spending.  From  FPDS-NG,  we 
judgmentally  selected  138  contracts  for  primarily  basic  services  coded  as 
performance-based  and  awarded  in  fiscal  years  2005  or  2006  at  Coast 
Guard,  CBP,  ICE,  and  TSA  with  obligations  greater  than  $1  million  dollars 
and  asked  the  components  to  verify  whether  these  contracts  included  the 
required  performance-based  elements  outlined  in  FAR  subpart  37.6.  Finally, 
we  interviewed  contracting  and  program  management  staff  and  DHS  Chief 
Procurement  Office  (CPO)  representatives  and  met  with  representatives 
from  the  Office  of  Management  and  Budget’s  Office  of  Federal 
Procurement  Policy  (OFPP).  To  evaluate  the  extent  to  which  DHS  used 
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different  contract  types  for  performance-based  acquisitions,  we  reviewed  the 
contract  type  on  all  contracts  DHS  verified  as  performance-based. 

We  conducted  this  performance  audit  from  January  2007  to  April  2008  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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US  Department  of  Homeland  Security 

Washington,  DC  20528 

tiSIf  Homeland 
If  Security 


April  11,2008 

Mr.  John  P.  Hutton 
Director 

Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Hutton: 

Thank  you  for  the  opportunity  to  review  and  comment  on  the  Government  Accountability 
Office's  (GAO’s)  draft  report  GAO-08-263  entitled  DEPARTMENT  OF  HOMELAND 
SECURITY:  Better  Planning  and  Assessment  Needed  to  Improve  Outcomes  for  Complex 
Service  Acquisitions.  Technical  comments  have  been  provided  under  separate  cover. 

The  Department  of  Homeland  Security  (DHS)  generally  concurs  with  the  findings  and 
recommendations  in  the  draft  report  and  offers  the  following  with  respect  to  the  report’s 
three  recommendations: 

GAO  Recommendation  #1:  Routinely  assess  requirements  for  major,  complex 
investments  to  ensure  they  are  well-defined  and  develop  consistently  measurable 
standards  linked  to  those  requirements. 

GAO  Recommendation  #2:  At  a  department-wide  level,  systematically  evaluate  the 
outcomes  of  major  investments  and  relevant  contracting  methods. 

DHS  Response  to  Recommendations  #1  and  #2:  DHS  is  in  the  midst  of  many  crucial 
acquisitions  that  are  vital  to  its  success.  The  Chief  Procurement  Officer  (CPO)  is 
working  to  strengthen  acquisition  and  procurement  by  institutionalizing  solid  processes, 
including  the  following  actions: 

A.  Strengthening  the  requirements  and  investment  review  processes  by  improving  the 
joint  requirements  council  and  Investment  Review  Board  (IRB)  process.  We  are 
preparing  to  initiate  a  new  Department-wide  requirements  process  and  have 
reinvigorated  our  investment  review  process; 

B.  Reviewing  the  major  programs  and  investments  to  ensure  that  the  requirements  are 
clear,  cost  estimates  are  valid,  technology  risks  are  properly  assessed,  schedules  are 
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realistic,  contract  vehicles  are  proper,  and  the  efforts  are  well  managed.  We  have 
held  one  formal  Deputy  Secretary  IRB  and  are  projecting  one  per  month.  We  are 
also  beginning  the  processes  to  conduct  paper  IRBs  and  Deputy  Undersecretary  for 
Management  IRBs,  as  well  as  establishing  Acquisition  Program  Baselines  (APBs) 
and  authorizing  execution  to  the  APB  for  all  Level  1  and  2  programs; 

C.  Building  the  capability  to  manage  complex  efforts  by  ensuring  that  program  offices 
are  properly  structured  and  staffed  with  the  right  people  and  skills  to  ensure 
efficient  and  effective  program  management  and  oversight;  and  to  aggressively  hire 
where  we  have  known  shortages;  and 

D.  Examining  best  practice  metrics  in  use  by  other  departments  with  the  intent  to  start 
implementation  this  year. 

The  Acquisition  Program  Management  Division  (APMD)  of  the  Office  of  the 
Chief  Procurement  Officer  (OCPO)  began  operations  in  August  2007.  The  division  was 
established  to  provide  oversight  and  support  for  acquisition  programs.  To  date,  APMD 
has  performed  Quick  Look  assessments  of  37  Level  1  programs  and  has  overseen  Deep 
Dive  reviews  of  the  SBInet  and  Advance  Spectroscopic  Portal  (ASP)  programs.  APMD 
has  provided  advice  and  guidance  to  a  number  of  programs,  particularly  in  the  area  of 
cost  benefit  analysis.  Currently  the  APMD  team  is  focused  on  an  aggressive  Investment 
&  Acquisition  process  re-engineering  effort.  The  effort  includes  replacing  DHS 
Management  Directive  1400  Investment  Review  Process,  establishing  revised  investment 
and  acquisition  decision  procedures,  as  well  as  processes  for  acquisition  program 
baselining,  periodic  reporting,  acquisition  of  services,  and  other  initiatives  as  they  are 
identified. 

CPO  is  also  working  to  assure  that  DHS  obtains  qualified  acquisition 
professionals  at  the  right  time  with  the  right  skill-set.  Competition  for  these  professionals 
is  intense  within  the  Washington,  D.C.  area.  To  resolve  our  personnel  shortages,  we  are 
intensifying  our  human  capital  planning  efforts  to  minimize  skill  and  competency  gaps  as 
well  as  minimize  our  critical  vacancies  and  reliance  on  contractors.  For  example,  in 
response  to  the  OMB  1102  Contracting  Workforce  Competency  Gap  Survey,  we 
developed  a  training  plan  that  spans  the  next  three  years.  This  training  plan  targets  the 
contracting  functional  area  within  the  DHS  Acquisition  Workforce,  but  it  will  also 
benefit  other  acquisition  career  fields  including  program  management  and  Contracting 
Officer's  Technical  Representatives. 

CPO  is  also  currently  conducting  staffing  studies  to  better  define  our  acquisition 
workforce  needs.  Currently  our  workforce  includes  program  managers  and  contract 
specialists.  As  part  of  our  human  capital  planning  efforts,  we  will  be  identifying  other 
required  acquisition  career  fields  such  as  test  and  evaluation,  systems  engineering, 
logistics,  and  cost  estimating.  We  are  aggressively  working  to  ensure  that  each 
acquisition  position,  upon  definition,  is  encumbered  by  an  acquisition  professional 
trained  and  certified  at  the  appropriate  level.  To  this  end,  we  are  continuously  reviewing 
and  updating  our  Acquisition  Training  Program,  the  underpinning  of  a  good  certification 
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program.  We  are  utilizing  the  Defense  Acquisition  Workforce  Improvement  Act 
framework  to  develop  DHS  certification  standards.  We  have  also  centralized  a  number 
of  recruiting  activities  including  issuing  Department-wide  vacancy  announcements.  Our 
centralized  recruitment  efforts  to  date  have  focused  primarily  on  contracting 
professionals.  Expansion  to  other  acquisition  career  fields  will  occur  as  each  series  is 
defined  and  Department-wide  needs  are  identified.  This  initiative  supplements  our 
Components'  on-going  recruitment  efforts  with  a  goal  of  recruiting  the  best  candidates 
available.  In  2005,  the  Department  commenced  the  Acquisition  Fellows  Program.  The 
goal  of  the  Fellows  Program  was  to  attract  new  talent  at  the  entry  level  into  our 
acquisition  positions,  and  retain  and  train  them  through  a  professional  career 
development  program.  Building  on  the  success  of  the  Acquisition  Fellows  Program,  CPO 
formalized  and  modeled  it  to  further  resemble  the  highly  successful  Department  of 
Defense  program.  This  year  CPO  received  funding  for  the  Acquisition  Professional 
Career  Program  and  33  of  the  requested  66  positions  were  funded.  In  FY  2009,  the  plan 
provides  for  a  total  of  100  positions  to  be  funded.  Our  inaugural  class  in  the  Acquisition 
Professional  Career  Program  began  in  January  2008  and  a  second  class  will  begin  in  June 
2008.  Our  goal  is  to  grow  this  program  to  300  positions  by  FY  2011  to  fill  our  critical 
acquisition  positions. 

GAO  Recommendation  #3:  Continuously  improve  the  quality  of  Federal  Procurement 
Data  System-Next  Generation  (FPDS-NG)  data  to  facilitate  the  ability  to  accurately 
identify  and  assess  the  use  and  outcomes  of  various  contracting  methods. 

DHS  Response  to  Recommendation  #3:  As  part  of  CPO’s  oversight  reviews,  the 
accuracy  of  the  FPDS  data  is  validated  for  the  review  sample,  including  whether  the 
contract  has  been  properly  coded  as  a  performance  based  contract.  In  addition,  OCPO  is 
an  active  member  of  the  Office  of  Federal  Procurement  Policy  group  that  is  working  to 
improve  FPDS.  OCPO  has  also  established  a  Governance  Board  whereby  CPO  reaches 
out  to  the  DHS  Components  to  improve  data  collection. 


Thank  you  again  for  the  opportunity  to  comment  on  this  draft  report  and  we  look  forward 
to  working  with  you  on  future  homeland  security  issues. 


Sincerely, 


y/ 


Penelope  G.  McCormack 
Acting  Director 

Departmental  Audit  Liaison  Office 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “E-mail  Updates.” 
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The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  DC  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 
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